Augmented reality device and method for detecting user&#39;s gaze

ABSTRACT

An augmented reality (AR) device for detecting a user&#39;s gaze and a method thereof are provided. The AR device includes a waveguide; a light reflector including a pattern; a support configured to fix the AR device to the user&#39;s face of the AR device; a light emitter and a light receiver installed on the support; and at least one processor, wherein the at least one processor is configured to control the light emitter to emit light toward the light reflector, identify the pattern based on the light received through the light receiver, and obtain gaze information of a user of the AR device based on the identified pattern.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a bypass continuation of International ApplicationNo. PCT/KR2022/000682 designating the United States, filed on Jan. 13,2022, in the Korean Intellectual Property Office and claiming priorityto Korean Patent Application No. 10-2021-0008942, filed on Jan. 21,2021, Korean Patent Application No. 10-2021-0084155, filed on Jun. 28,2021, and Korean Patent Application No. 10-2021-0128345, filed on Sep.28, 2021, in the Korean Intellectual Property Office, the disclosures ofwhich are incorporated by reference herein in their entireties.

TECHNICAL FIELD

The disclosure relates to an augmented reality (AR) device and methodfor detecting a user's gaze, and more particularly, to an AR device fordetecting a user's gaze and a method thereof by using a light emitterand a light receiver located in a support of the AR device.

BACKGROUND ART

Augmented reality (AR) is a technology that projects a virtual imageonto a physical environment space of the real world or a real worldobject and displays the virtual image as a single image. While beingworn on a user's face or head, an AR device allows the user to see areal scene and a virtual image through a glasses type device using asee-through display such as a waveguide in front of the user's eyes. Asresearch on such an AR device is being actively conducted, various typesof wearable devices have been released or are expected to be released.In the glasses type AR device of the related art, a camera is generallyarranged on a rim portion surrounding a waveguide to track the user'sgaze, which causes the rim portion of the AR device to be enlarged, andfurther, causes the user wearing the AR device to feel uncomfortable.

DESCRIPTION OF EMBODIMENTS Technical Problem

Provided are an augmented reality (AR) device and a method capable ofdetecting a user's gaze by using a light reflector and a light receiverlocated in a support extending from a frame of the AR device.

Provided are an AR device and a method capable of detecting a user'sgaze by using light reflected through a light reflector formed on awaveguide.

Provided are an AR device and a method capable of more accuratelydetecting a user's gaze by calculating a degree of bias of a support ofthe AR device based on a pattern formed on a light reflector.

Solution to Problem

According to an aspect of the disclosure, there is provided an augmentedreality (AR) device including: a waveguide; a light reflector comprisinga pattern; a support configured to fix the AR device to a user's face ofthe AR device; a light emitter and a light receiver installed on thesupport; and at least one processor configured to: control the lightemitter to emit light toward the light reflector, identify the patternbased on the light received by the light receiver, and obtain gazeinformation of a user of the AR device based on the identified pattern,wherein the light emitted toward the light reflector is reflected by thelight reflector and directed toward an eye of the user, and wherein thelight received by the light receiver comprises light from the lightdirected toward the eye of the user being reflected by the eye of theuser.

The support may include a temple extending from a frame around thewaveguide to be positioned on an ear of the user; and a nose supportextending from the frame and positioned on a nose of the user.

The light reflector may be coated on the waveguide.

The light reflector may be formed on the waveguide.

The at least one processor may be further configured to analyze theidentified pattern and identify a degree of bias of the support withrespect to the frame, the support extending from the frame.

The at least one processor may be further configured to: generate amapping function for calculating a position of a gaze point of the userbased on the degree of bias of the support with respect to the frame,and based on the mapping function and the degree of bias of the supportwith respect to the frame, obtain the gaze information of the user.

The at least one processor may be further configured to, based on thelight received by the light receiver, obtain a position of one or morefeature points corresponding to the eye of the user.

The at least one processor may be configured to input the position ofthe one or more feature points corresponding to the eye of the user andthe degree of bias of the support with respect to the frame into themapping function and calculate the position of the gaze point of theuser.

The position of the one or more feature points may include a position ofa pupil feature point of the eye of the user and a position of a glintfeature point of the eye of the user.

The at least one processor may be further configured to: display atarget point at a specific position on the waveguide in order tocalibrate the mapping function, receive light reflected by the eye ofthe user looking at the displayed target point through the lightreceiver, and calibrate the mapping function based on the lightreflected by the eye of the user looking at the displayed target point.

The at least one processor may be further configured to: based on thelight reflected by the eye of the user looking at the displayed targetpoint, identify the pattern of the light reflector, based on theidentified pattern, identify the degree of bias of the support, andbased on the light reflected by the eye of the user looking at thedisplayed target point, obtain a position of one or more feature pointscorresponding to the eye of the user looking at the displayed targetpoint.

The at least one processor may be further configured to input a degreeof bias of the temple and the position of the one or more feature pointsinto the mapping function, and calibrate the mapping function so that aposition value of the target point is output from the mapping function.

The light emitter may be an infrared light-emitting diode (IR LED), andthe light receiver is an IR camera.

The light emitter may be an infrared (IR) scanner, and the lightreceiver is an IR detector.

The at least one processor may be further configured to: based on IRlight obtained from the IR detector, obtain a position of one or morefeature points corresponding to the eye of the user calibrated accordingto a degree of bias of the support with respect to the frame, and obtainthe gaze information of the user based on the calibrated position of theone or more feature points corresponding to the eye of the user.

According to another aspect of the disclosure, there is provided amethod, performed by an augmented reality (AR) device, of detecting auser's gaze, the method including: emitting, by a light emitterinstalled in a support of the AR device, light toward a light reflectorcomprising a pattern, the light emitted by the light emitter beingdirected toward an eye of a user wearing the AR device, receiving, by alight receiver installed on the support, the light reflected by the eyeof the user, identifying the pattern based on the light received throughthe light receiver, and obtaining gaze information of the user based onthe identified pattern.

The method may further include analyzing the identified pattern andidentifying a degree of bias of the support with respect to the framebased on the identified pattern, wherein the support extends from theframe, wherein the obtaining of the gaze information may includedetermining a gaze direction of the user based on the degree of bias ofthe support with respect to the frame.

The method may further include generating a mapping function forcalculating a position of a gaze point of the user based on the degreeof bias of the support with respect to the frame, wherein the obtainingof the gaze information may include calculating the position of the gazepoint of the user based on the mapping function and the degree of biasof the support with respect to the frame.

The method may further include, based on the light received by the lightreceiver, obtaining a position of one or more feature pointscorresponding to the eye of the user, wherein the obtaining of the gazeinformation comprises inputting the position of the one or more featurepoints corresponding to the eye of the user and the degree of bias ofthe support with respect to the frame into the mapping function andcalculating the position of the gaze point of the user.

According to another aspect of the disclosure, there is provided acomputer-readable recording medium having recorded thereon a program forexecuting a method including: emitting, by a light emitter installed ina support of the AR device, light toward a light reflector comprising apattern, the light emitted by the light emitter being directed toward aneye of a user wearing the AR device; receiving, by a light receiverinstalled on the support, the light reflected by the eye of the user;identifying the pattern based on the light received through the lightreceiver; and obtaining gaze information of the user based on theidentified pattern.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example in which an augmentedreality (AR) device detects a user's gaze using a gaze detector locatedin a temple portion of the AR device, according to an example embodimentof the disclosure.

FIG. 2 is a diagram illustrating an example of an AR device according toan example embodiment of the disclosure.

FIG. 3 is a block diagram of an AR device according to an exampleembodiment of the disclosure.

FIG. 4 is a diagram illustrating an example of operations of a lightemitter and a light receiver of an AR device, according to an exampleembodiment of the disclosure.

FIG. 5A is a diagram illustrating an example of a light emitter thatemits planar light, according to an example embodiment of thedisclosure.

FIG. 5B is a diagram illustrating an example of a light emitter thatemits point light, according to an example embodiment of the disclosure.

FIG. 5C is a diagram illustrating an example of a light emitter thatemits line light, according to an example embodiment of the disclosure.

FIG. 6A is a diagram illustrating an example in which a light emitterand a light receiver are arranged in a temple of an AR device, accordingto an example embodiment of the disclosure.

FIG. 6B is a diagram illustrating an example in which a light emitterand a light receiver are arranged in a nose support of an AR device,according to an example embodiment of the disclosure.

FIG. 6C is a diagram illustrating an example in which a light emitterand a light receiver are arranged in a temple and a nose support of anAR device, according to an example embodiment of the disclosure.

FIG. 6D is a diagram illustrating an example in which a light emitterand a light receiver are arranged in a temple and a nose support of anAR device, according to an example embodiment of the disclosure.

FIG. 7A is a diagram illustrating an example of a dot pattern formed ona light reflector of an AR device, according to an example embodiment ofthe disclosure.

FIG. 7B is a diagram illustrating an example of a grid pattern formed ona light reflector of an AR device, according to an example embodiment ofthe disclosure.

FIG. 7C is a diagram illustrating an example of a pattern in the form ofa 2D marker, according to an example embodiment of the disclosure.

FIG. 7D is a diagram illustrating an example of a light reflector thatcovers a part of a waveguide, according to an example embodiment of thedisclosure.

FIG. 8A is a diagram illustrating a light emission angle and a patternbefore a temple of an AR device is biased, according to an exampleembodiment of the disclosure.

FIG. 8B is a diagram illustrating a light emission angle and a patternafter a temple of an AR device, is biased according to an exampleembodiment of the disclosure.

FIG. 9 is a diagram illustrating an example of a pattern identified froman array of light received through a light receiver when a light emitterof an AR device is an infrared (IR) scanner, according to an exampleembodiment of the disclosure.

FIG. 10 is a diagram illustrating an example of an eye featureidentified from an array of light received through a light receiver ofthe AR device when a light emitter of an AR device is an IR scanner,according to an example embodiment of the disclosure.

FIG. 11 is a diagram illustrating examples of functions used by an ARdevice to calculate a center of an eyeball and calculate a gaze point ofa user, according to an example embodiment of the disclosure.

FIG. 12 is a flowchart of a method, performed by an AR device, ofdetecting a user's gaze, according to an example embodiment of thedisclosure.

MODE OF DISCLOSURE

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Hereinafter, embodiments of the disclosure will now be described indetail with reference to the accompanying drawings for one of skill inthe art to be able to perform the disclosure without any difficulty. Thedisclosure may, however, be embodied in many different forms and shouldnot be construed as being limited to the embodiments of the disclosureset forth herein. In order to clearly describe the disclosure, portionsthat are not relevant to the description of the disclosure are omitted,and similar reference numerals are assigned to similar elementsthroughout the present specification.

Throughout the specification, it will be understood that when an elementis referred to as being “connected to” another element, it may be“directly connected to” the other element or be “electrically connectedto” the other element through an intervening element. In addition, whenan element is referred to as “including” a constituent element, otherconstituent elements may be further included not excluded unless thereis any other particular mention on it.

The term ‘augmented reality (AR)’ herein denotes a technology thatprovides viewing of a virtual image on a physical environment space ofthe real world or viewing of a virtual image together with a realobject.

In addition, the term ‘AR device’ denotes a device capable of creating‘AR’, and includes not only AR glasses that are typically worn on auser's face but also includes head-mounted display (HMD) apparatuses andAR helmets that are worn on the user's head, etc.

Meanwhile, the term ‘real scene’ denotes a scene of the real world thatthe user sees through the AR device, and may include a real worldobject. In addition, the term ‘virtual image’ denotes an image generatedby an optical engine, and may include both a static image and a dynamicimage. The virtual image may be observed with a real scene, and may bean image representing information about a real object in the real scene,information about an operation of the AR device, a control menu, etc.

Accordingly, an AR device may be equipped with an optical engine togenerate a virtual image including light generated by a light source,and a waveguide formed of a transparent material to guide the virtualimage generated by the optical engine to the user's eyes and allow theuser to see a scene of the real world together with the virtual image.In addition, as described above, the AR device needs to be able to allowthe user to observe a scene of the real world, and thus, an opticalelement for redirecting the path of light that basically hasstraightness is required in order to guide the light generated by theoptical engine to the user' eyes through the waveguide. Here, the pathof the light may be redirected by using reflection by, for example, amirror, or by using diffraction by a diffractive element, for example, adiffractive optical element (DOE) or a holographic optical element(HOE), but the disclosure is not limited thereto.

Hereinafter, the disclosure will be described in detail with referenceto the accompanying drawings.

FIG. 1 is a diagram illustrating an example in which an augmentedreality (AR) device 1000 detects a user's gaze using a gaze detector1500 located in a temple portion of the AR device 1000 according to anexample embodiment of the disclosure.

Referring to FIG. 1, the AR device 1000 may detect the user's gaze byusing a light emitter 1510 and a light receiver 1520. The light emitter1510 and the light receiver 1520 used to detect the user's gaze may beprovided in, for example, the temple portion of the AR device 1000, andthe AR device 1000 may effectively identify user's eyes by using thelight emitter 1510 and the light receiver 1520 provided in the templeportion. Infrared (IR) light may be emitted from the light emitter 1510provided in the temple portion toward a waveguide of the AR device 1000,reflected by a light reflector, and received from the user's eyesthrough the light receiver 1520 provided in the temple portion. Also,the AR device 1000 may obtain information about the user's eyes based onthe received IR light, and detect a gaze direction of the user by usingthe obtained information about the eyes.

The AR device 1000 denotes a device capable of creating ‘AR’, and mayinclude, for example, AR glasses that are worn on a user's face, but thedisclosure is not limited thereto. For example, the AR device 1000 mayinclude a head-mounted display (HMD) apparatus and an AR helmet that areworn on the user's head, etc. In this case, a gaze detector 1500 may beprovided on an inner side part of the HMD apparatus facing the side ofthe user's eyes in the HMD apparatus or on an inner side part of the ARhelmet facing the side of the user's eyes in the AR helmet.

FIG. 2 is a diagram illustrating an example of the AR device 1000according to an example embodiment of the disclosure.

Referring to FIG. 2, the AR device 1000 may include a glasses-type bodyconfigured to be worn by a user as a glasses-type display device.

The glasses-type body may include a frame 110 and a support 190. Thesupport 190 may extend from the frame 110 and be used to seat the ARdevice 1000 on a user's head. The support 190 may include a temple 191and a nose support 192. The temple 191 may extend from the frame 110 andmay be used to fix the AR device 1000 to the user's head on a sidesurface of the glasses-type body. The nose support 192 may extend fromthe frame 110 and may be used to seat the AR device 1000 on a user'snose, and may include, for example, a nose bridge and a nose pad, butthe disclosure is not limited thereto.

Also, a waveguide 170 to which a light reflector 1400 is attached may belocated on the frame 110. The frame 110 may be formed to surround anouter circumferential surface of the waveguide 170. The waveguide 170may be configured to receive projected light in an input region andoutput at least part of the input light in an output region. Thewaveguide 170 may include a left eye waveguide 170L and a right eyewaveguide 170R.

A left eye light reflector 1400L and the left eye waveguide 170L may beprovided at positions corresponding to a user's left eye, and a righteye light reflector 1400R and the right eye waveguide 170R may beprovided at positions corresponding to a user's right eye. For example,the left eye light reflector 1400L may be attached to the left eyewaveguide 170L, or the right eye light reflector 1400R may be attachedto the right eye waveguide 170R, but the disclosure is not limitedthereto. In addition, for example, the left eye light reflector 1400Lmay be coated on the inner side of the left eye waveguide 170L to beattached to the left eye waveguide 170L, or the right eye lightreflector 1400R may be coated on the inner side of the right eyewaveguide 170R to be attached to the right eye waveguide 170R.

In addition, an optical engine 120 of a projector that projects displaylight including an image may include a left eye optical engine 120L anda right eye optical engine 120R. The eye optical engine 120L and theright eye optical engine 120R may be located on both sides of the ARdevice 1000. Alternatively, one optical engine 120 may be included in acentral portion around the nose support 192 of the AR device 1000. Lightemitted from the optical engine 120 may be displayed through thewaveguide 170.

The light emitter 1510 and the light receiver 1520 of the gaze detector1500 may be provided on an inner side part of the support 190 of the ARdevice 1000, which is a position between the support 190 and user'seyes. The light emitter 1510 and the light receiver 1520 may be providedto face the light reflector 1400 in the support 190 of the AR device1000. For example, the light emitter 1510 and the light receiver 1520may be provided at positions spaced from the frame 110 by about 10 mm to15 mm on the inner side of the temple 191 of the AR device 1000, inorder to respectively emit and receive IR light without being disturbedby user's hair, etc.

FIG. 3 is a block diagram of the AR device 1000 according to an exampleembodiment of the disclosure.

Referring to FIG. 3, the AR device 1000 according to an exampleembodiment of the disclosure may include a user inputter 1100, amicrophone 1200, a display 1300, a light reflector 1400, a gaze detector1500, a communication interface 1600, a storage 1700, and a processor1800. Also, the gaze detector 1500 may include the light emitter 1510and the light receiver 1520.

The user inputter 1100 refers to a means by which a user inputs data forcontrolling the AR device 1000. For example, the user inputter 1100 mayinclude a key pad, a dome switch, a touch pad (e.g., a touch-typecapacitive touch pad, a pressure-type resistive overlay touch pad, aninfrared sensor-type touch pad, a surface acoustic wave conduction touchpad, an integration-type tension measurement touch pad, a piezoelectriceffect-type touch pad), a jog wheel, a jog switch, but the disclosure isnot limited thereto.

The microphone 1200 may receive an external audio signal, and processthe received audio signal into electrical voice data. For example, themicrophone 1200 may receive an audio signal from an external device or aspeaker. The microphone 1200 may use various denoising algorithms forremoving noise generated during a process of receiving the externalaudio signal. The microphone 1200 may receive a voice input of the userfor controlling the AR device 1000.

The display 1300 may display information processed by the AR device1000. For example, the display 1300 may display a user interface forcapturing an image of surroundings of the AR device 1000, andinformation related to a service provided based on the captured image ofthe surroundings of the AR device 1000.

According to an example embodiment of the disclosure, the display 1300may provide an AR image. The display 1300 according to an exampleembodiment of the disclosure may include the waveguide 170 and theoptical engine 120. The waveguide 170 may include a transparent materialthrough which a partial region of a rear surface is visible when theuser wears the AR device 1000. The waveguide 1320 may be configured as aflat plate of a single layer or multi-layer structure including atransparent material through which light may be internally reflected andpropagated. The waveguide 1320 may face an exit surface of the opticalengine 120 to receive light of a virtual image projected from theoptical engine 120. Here, the transparent material is a material throughwhich light is capable of passing, its transparency may not be 100%, andmay have a certain color. According to an example embodiment of thedisclosure, the waveguide 170 includes the transparent material, andthus the user may view not only a virtual object of the virtual imagebut also an external real scene, so that the waveguide 170 may bereferred to as a see-through display. The display 1300 may output thevirtual object of the virtual image through the waveguide 170, therebyproviding an AR image. When the AR device 1000 is a glasses type device,the display 1300 may include a left display and a right display.

The light reflector 1400 may reflect light emitted from the lightemitter 1510 which will be described later. The light reflector 1400 andthe waveguide 170 may be provided at positions facing the user's eyes,and may be attached to each other. For example, the light reflector 1400may be coated on at least a partial region of the waveguide 170. Inaddition, the light reflector 1400 may be attached to or coated on otherelements included in the glasses type AR device 1000 in addition to thewaveguide 170, for example, a vision correcting lens for visioncorrection or a cover glass installed to protect the waveguide 170. Thelight reflector 1400 may include a material capable of reflecting IRlight emitted from the light emitter 1510. The light reflector 1400 maybe, for example, silver, gold, copper, ora material including one ormore of these metals, but the disclosure is not limited thereto.Accordingly, the IR light emitted from the light emitter 1510 may bereflected by the light reflector 1400 and directed toward the user'seyes, and the IR light reflected back from the user's eyes may bereflected by the light reflector 1400 and directed toward the lightreceiver 1520.

The light reflector 1400 may be coated on the waveguide 170 to have acertain pattern. The pattern formed on the light reflector 1400 mayinclude, for example, a dot pattern, a line pattern, a grid pattern, a2D marker, etc., but the disclosure is not limited thereto. In addition,the pattern formed on the light reflector 1400 may be formed on, forexample, a part of the waveguide 170 at which the user's gaze is lessfrequently directed. The pattern formed on the light reflector 1400 maybe formed on, for example, a part of the waveguide 170 that does notinterfere with capturing or scanning the user's eyes. For example, thecertain pattern may indicate a pattern formed by a part in which lightemitted from the light emitter 1510 is reflected and a part in which thelight is not reflected, in the light reflector 1400. Because lightemitted toward the part in which the light is not reflected, in thelight reflector 1400 is not reflected by the light reflector 1400, thelight receiver 1520 does not receive the light emitted toward the partin which the light is not reflected. Accordingly, the pattern of thelight reflector 1400 formed by the part in which light is reflected andthe part in which the light is not reflected may be detected from thelight received by the light receiver 1520. In addition, when the lightemitted from the light emitter 1510 is IR light, the certain pattern mayinclude a material for reflecting the IR light, and the material forreflecting the IR light may not be visible to the user's eyes. Becausemost of a real world light or real scene observed by the user throughthe AR device 1000 includes visible light, the user may not be disturbedby the light reflector 1400 in which the certain pattern is formed, andmay observe the real world light or real scene.

The gaze detector 1500 may include the light emitter 1510 that emits IRlight for detecting the user's gaze and the light receiver 1520 thatreceives the IR light, and may detect data related to the user's gaze ofthe user who wears the AR device 1000.

The light emitter 1510 of the gaze detector 1500 may emit the IR lighttoward the light reflector 1400 so that the IR light reflected by thelight reflector 1400 may be directed toward the user's eyes. The lightemitter 1510 may emit the IR light toward the light reflector 1400, theemitted IR light may be reflected by the light reflector 1400, and thereflected IR light may be directed toward the user's eyes. The lightemitter 1510 may be provided at a position in the AR device 1000 wherethe IR light may be emitted toward the light reflector 1400. The lightemitter 1510 may be located on the support 190 of FIG. 2 that supportsthe AR device 1000 on the user's face, for example, like the temple 191and the nose support 192 of FIG. 2.

In addition, the IR light reflected from the user's eyes may bereflected by the light reflector 1400 and received by the light receiver1520 of the gaze detector 1500. The IR light directed toward the user'seyes may be reflected from the user's eyes, the IR light reflected fromthe user's eyes may be reflected by the light reflector 1400, and thelight receiver 1520 may receive the IR light reflected by the lightreflector 1400. The light receiver 1520 may be provided at a position inthe AR device 1000 where the IR light reflected from the light reflector1400 may be received. The light receiver 1520 may be located on thesupport 190 of FIG. 2 that supports the AR device 1000 on the user'sface, for example, like the temple 191 and the nose support 192 of FIG.2. Also, for example, the nose support 192 of FIG. 2 may include a nosebridge and a nose pad. In addition, the nose bridge and the nose pad maybe integrally configured, but the disclosure is not limited thereto.

For example, the light emitter 1510 may be an IR light-emitting diode(LED) that emits IR light, and the light receiver 1520 may be an IRcamera that captures IR light. In this case, the IR camera may capturethe user's eyes using the IR light reflected by the light reflector1400. When the light emitter 1510 is the IR LED and the light receiver1520 is the IR camera, the light emitter 1510 may emit IR light ofplanar light toward the light reflector 1400, and the light receiver1520 may receive the IR light of the planar light reflected from thelight reflector 1400. The planar light may be light emitted in a planarform, and the planar light emitted from the light emitter 1510 may bedirected toward at least a part of the entire region of the lightreflector 1400. At least a part of the entire region of the lightreflector 1400 may be set so that the planar light reflected from atleast a part of the entire region of the light reflector 1400 may coverthe user's eyes.

Alternatively, for example, the light emitter 1510 may be an IR scannerthat emits IR light, and the light receiver 1520 may be an IR detectorthat detects IR light. In this case, the IR scanner may emit IR light sothat the IR light for scanning the user's eyes is directed toward theuser's eyes, and the IR detector may detect the IR light reflected fromthe user's eyes. When the light emitter 1510 is the IR scanner thatemits IR light and the light receiver 1520 is the IR detector thatdetects IR light, the light emitter 1510 may emit line lights in theform of line, and the line lights emitted from the light emitter 1510may be directed toward a part of the entire region of the lightreflector 1400. At least a part of the entire region of the lightreflector 1400 may be set so that the line lights reflected from atleast a part of the entire region of the light reflector 1400 may coverthe user's eyes. When the light emitter 1510 is the IR scanner thatemits IR light and the light receiver 1520 is the IR detector thatdetects IR light, the light emitter 1510 may emit point lights in theform of point, and the point lights emitted from the light emitter 1510may be directed toward a part of the entire region of the lightreflector 1400. At least a part of the entire region of the lightreflector 1400 may be set so that the point lights reflected from atleast a part of the entire region of the light reflector 1400 may coverthe user's eyes.

When the light emitter 1510 is the IR scanner and the light receiver1520 is the IR detector, the light emitter 151 may emit IR light of thepoint light or the line light to the light reflector 1400, and the lightreceiver 1520 may receive the IR light of the point light or the linearlight reflected from the light reflector 1400. In this case, the lightemitter 1510 may sequentially emit IR light while moving a lightemitting direction of the light emitter 1510 so that the IR light of thepoint light or the line light may cover a space where the user's eyesare located. Although the IR scanner generally includes the IR LED and amicro-electro mechanical systems (MEMS) mirror capable of controlling adirection of the IR light emitted from the IR LED and reflecting the IRlight, hereinafter, the IR scanner, the IR LED and the MEMS mirror arecollectively referred to as and described as an IR scanner. In addition,although the IR detector generally includes several photodiodesinstalled in a part where light detection is required, hereinafter, theIR detector and photodiodes are described as the IR detector.

When the AR device 1000 is a glasses type device, the light emitter 1510and the light receiver 1520 may be provided on the temple 191 of the ARdevice 1000. For example, referring to FIG. 2, the light emitter 1510and the light receiver 1520 may be provided on an inner side part of thetemple 191 of the AR device 1000, which is a position between the temple191 and user's eyes. For example, referring to FIG. 2, the light emitter1510 and the light receiver 1520 may be provided at positions spacedfrom the frame 110 by about 10 mm to 15 mm on the inner side of thetemple 191 of the AR device 1000 The light emitter 1510 and the lightreceiver 1520 may be provided to face the light reflector 1400 in thetemple 191 of the AR device 1000.

Also, for example, referring to FIG. 2, the light emitter 1510 and thelight receiver 1520 may be provided on the nose support 192 of the ARdevice 1000. The light emitter 1510 and the light receiver 1520 may beprovided on an inner side part of the nose support 192 of the AR device1000, which is a position between the nose support 192 and the user'seyes. For example, referring to FIG. 2, the light emitter 1510 and thelight receiver 1520 may be provided at positions spaced from the frame110 by about 10 mm to 15 mm on an inner side of the nose support 192 ofthe AR device 1000. The light emitter 1510 and the light receiver 1520may be provided to face the light reflector 1400 in the nose support 192of the AR device 1000.

The gaze detector 1500 may provide data related to the gaze of theuser's eyes to the processor 1800, and the processor 1800 may obtaingaze information of the user based on the data related to the gaze ofthe user's eyes. The data related to the gaze of the user's eyes is dataobtained by the gaze detector 1500, and may include a type (e.g., pointlight, line light, or planar light) of IR light emitted from the lightemitter 1510, characteristics of the IR light emitted from the lightemitter 1510, data regarding an emission region of the IR light emittedfrom the light emitter 1510, and data indicating the characteristics ofthe IR light received from the light receiver 1520. Further, the gazeinformation of the user is information related to the user's gaze, maybe generated by analyzing the data related to the gaze of the user'seyes, and may include information about, for example, a location of theuser's pupil, a location of a pupil central point, a location of theuser's iris, a center of the user's eyes, a location of glint featurepoint of the user's eyes, a gaze point of the user, a gaze direction ofthe user, etc. but the disclosure is not limited thereto. The gazedirection of the user may be, for example, a direction of eth user'sgaze from the center of the user's eyes toward the gaze point at whichthe user gazes. For example, the gaze direction of the user may berepresented by a vector value from the center of the user's left eyetoward the gaze point and a vector value from the center of the user'sright eye toward the gaze point, but the disclosure is not limitedthereto. According to an example embodiment of the disclosure, the gazedetector 1500 may detect data related to the gaze of the user wearingthe AR device 1000 at a previously determined time interval.

The communication interface 1600 may transmit/receive data for receivinga service related to the AR device 1000 to/from an external device and aserver.

The storage 1700 may store a program to be executed by the processor1800, which will be described later, and may store data input to oroutput from the AR device 1000.

The storage 1700 may include at least one of an internal memory or anexternal memory. The internal memory may include, for example, at leastone of a volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM),synchronous dynamic RAM (SDRAM), etc.), a non-volatile memory (e.g., onetime programmable ROM (OTPROM), programmable ROM (PROM), erasable andprogrammable ROM (EPROM), electrically erasable and programmable ROM(EEPROM), mask ROM, flash ROM, etc.), hard disk drive (HDD), or solidstate drive (SSD). According to an example embodiment of the disclosure,the processor 1800 may load a command or data received from at least oneof the non-volatile memory or another element into the volatile memoryand process the command or the data. Also, the processor 1800 may storedata received or generated from another element in the non-volatilememory. The external memory may include, for example, at least one ofcompact flash (CF), secure digital (SD), micro secure digital(Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), ormemory stick.

Programs stored in the storage 1700 may be classified into a pluralityof modules according to their functions. According to an exampleembodiment, each of the plurality of modules may include one or morecomputer readable codes, which may include, for example, a lightirradiation code 1710, a light reception code 1720, an eye featuredetection code 1730, a pattern detection code 1740, a bias determinationcode 1750, a pupil position detection code 1760, a gaze determinationcode 1770, and a calibration code 1780. For example, a memory may beincluded in the gaze detector 1500, and, in this case, the lightirradiation code 1710 and the light reception code 1720 may be stored asfirmware in the memory included in the gaze detector 1500.

The processor 1800 controls the overall operation of the AR device 1000.For example, the processor 1800 may execute the programs stored in thestorage 1700, thereby generally controlling the user inputter 1100, themicrophone 1200, the display 1300, the light reflector 1400, the gazedetector 1500, the communication interface 1600, the storage 1700, etc.

The processor 1800 may execute the light irradiation code 1710, thelight reception code 1720, the eye feature detection code 1730, thepattern detection code 1740, the bias determination code 1750, the pupilposition detection code 1760, the gaze determination code 1770, and thecalibration code 1780 that are stored in the storage 1700, therebydetermining the gaze point of the user and the gaze direction.

According to an example embodiment of the disclosure, the AR device 1000may include a plurality of processors 1800, and the light irradiationcode 1710, the light reception code 1720, the eye feature detection code1730, the pattern detection code 1740, the bias determination code 1750,the pupil position detection code 1760, the gaze determination code1770, and the calibration code 1780 may be executed by the plurality ofprocessors 1800.

For example, some of the light irradiation code 1710, the lightreception code 1720, the eye feature detection code 1730, the patterndetection code 1740, the bias determination code 1750, the pupilposition detection code 1760, the gaze determination code 1770, and thecalibration code 1780 may be executed by a first processor, and theothers of the light irradiation code 1710, the light reception code1720, the eye feature detection code 1730, the pattern detection code1740, the bias determination code 1750, the pupil position detectioncode 1760, the gaze determination code 1770, and the calibration code1780 may be executed by a second processor, but the disclosure is notlimited thereto.

For example, the gaze detector 1500 may include another processor and amemory, and the other processor may execute the light irradiation code1710 and the light reception code 1720 that are stored in the memory,and the processor 1800 may execute the eye feature detection code 1730,the pattern detection code 1740, the bias determination code 1750, thepupil position detection code 1760, the gaze determination code 1770,and the calibration code 1780 that are stored in the storage 1700.

The processor 1800 may execute the light irradiation code 1710 stored inthe storage 1700 so that the light emitter 1510 may emit IR light towardthe light reflector 1400. The processor 1800 may control the lightemitter 1510 by executing the light irradiation code 1710, and the lightemitter 1510 controlled by the processor 1800 may emit the IR lighttoward at least a partial region of the light reflector 1400 so that theIR light reflected by the light reflector 1400 may cover the user'seyes.

For example, when the light receiver 1520 is an IR camera, the lightemitter 1510 may be an IR LED, and the processor 1800 may control the IRLED so that the IR light emitted from the IR LED may be reflected by thelight reflector 1400 and irradiated to a region including the user'seyes, in order for the IR camera to capture the user's eyes. Forexample, in order to reflect the light emitted from the IR LED by usingthe light reflector 1400 and irradiate the light to the region includingthe user's eyes, the processor 1800 may control an irradiation directionof the IR light emitted from the IR LED, and apply power to the IR LED,thereby controlling emission of the IR light from the IR LED.

According to one example embodiment of the disclosure, the IR camera andthe IR LED may be installed toward the light reflector 1400 of the ARdevice 1000 so that the IR camera may capture the entire region of theuser's eyes, and the processor 1800 may control the IR LED installedtoward the light reflector 1400 to emit the IR light. An example inwhich the irradiation direction of the IR light emitted from the IR LEDis controlled will be described in more detail with reference to FIG.5A.

According to another example embodiment of the disclosure, when thelight receiver 1520 is an IR detector, the light emitter 1510 may be anIR scanner, and the processor 1800 may control the IR scanner to scanthe user's eyes by reflecting the IR light emitted from the IR scannerby using the light reflector 1400, so that the IR detector may detectthe user's eyes. For example, in order to scan the user's eyes byreflecting the light emitted from the IR scanner by using the lightreflector 1400, the processor 1800 may control an irradiation directionof the IR light emitted from the IR scanner, and apply power to the IRscanner, thereby controlling emission of the IR light from the IRscanner. An example in which the irradiation direction of the IR lightemitted from the IR scanner is controlled will be described in moredetail with reference to FIGS. 5B and 5C.

The processor 1800 may execute the light reception code 1720 stored inthe storage 1700 so that the light receiver 1520 may receive the lightreflected by the light reflector 1400 from the user's eyes. Theprocessor 1800 may control the light receiver 1520 by executing thelight reception code 1720, and the light receiver 1520 controlled by theprocessor 1800 may receive the light reflected by the light reflector1400 from the user's eyes.

For example, when the light emitter 1510 is an IR LED, the lightreceiver 1520 may be an IR camera, and the processor 1800 may controlthe IR camera to capture the user's eyes through the light reflected bythe light reflector 1400 from the user's eyes.

Alternatively, for example, when the light emitter 1510 is an IRscanner, the light receiver 1520 may be an IR detector, and theprocessor 1800 may control the IR detector to detect the IR lightreflected by the light reflector 1400 from the user's eyes, so that theIR detector may detect the user's eyes.

The processor 1800 may execute the eye feature detection code 1730stored in the storage 1700, thereby detecting features related to thegaze of the user's eyes. For example, the processor 1800 may execute theeye feature detection code 1730, thereby detecting a position of a pupilfeature point of the user's eyes and a position of a glint feature pointof the user's eyes. The pupil feature point may be, for example, a pupilcentral point, and the glint feature point of the eyes may be a parthaving brightness greater than or equal to a certain value in a detectedeye region. The position of the pupil feature point and the position ofthe glint feature point of the eyes may be identified, for example, by acoordinate value indicating a position in a coordinate system of thelight receiver 1520. For example, the coordinate system of the lightreceiver 1520 may be a coordinate system of an IR camera or a coordinatesystem of the IR detector, and the coordinate value in the coordinatesystem of the light receiver 1520 may be a 2D coordinate value.

The processor 1800 may detect features related to the gaze of the eyesby analyzing the light received by the light receiver 1520. For example,when the light receiver 1520 is an IR camera, the processor 1800 mayidentify the position of the pupil feature point and the position of theglint feature point of the eyes in an image captured by the IR camera.Alternatively, for example, when the light receiver 1520 is an IRdetector, the processor 1800 may analyze the IR light detected by the IRdetector, thereby identifying the position of the pupil feature pointand the position of the glint feature point of the eyes. When thepositions of the feature points are identified based on the IR lightdetected by the IR detector, the position of the pupil feature point andthe position of the glint feature point may have values calibrated byreflecting the bias of the support 190. The position of the pupilfeature point and the position of the glint feature point calibrated byreflecting the bias of the support 190 will be described in more detailwith reference to FIG. 11.

Also, the processor 1800 may analyze the light received by the lightreceiver 1520, thereby obtaining a coordinate value indicating theposition of the pupil feature point and a coordinate value indicatingthe position of the glint feature point of the eyes. For example, whenthe light receiver 1520 is an IR camera, the processor 1800 may obtainthe coordinate value of the pupil feature point and the coordinate valueof the glint feature point of the eyes from the coordinate system of theIR camera. The coordinate system of the IR camera may be used toindicate the position of the pupil feature point and the position of theglint feature point of the eyes, and, for example, coordinate valuescorresponding to pixels of an image captured by the IR camera on thecoordinate system of the IR camera may be previously set. Also, based ona property (e.g., brightness) of IR light received through the IRcamera, a coordinate value corresponding to a feature point of the eyesmay be identified.

For example, when the light receiver 1520 is an IR camera, the processor1800 may identify the position of the pupil central point in the imagecaptured by the IR camera. The processor 1800 may identify thebrightness of IR light received through an image sensor of the IR cameraincluding a plurality of photodiodes, and identify at least one pixelthat receives IR light indicating the pupil among the pixels of theimage captured by the IR camera, thereby identifying the position of thepupil central point. For example, positions of the pixels in the imagecaptured by an IR camera may be identified through the coordinate systemof the IR camera, and the position of the pupil central point may have acoordinate value in the coordinate system of the IR camera, as aposition value of at least one pixel corresponding to the pupil centralpoint.

For example, the processor 1800 may identify a position of the brightestpoint in the image captured by the IR camera, in order to identify theglint feature point of the eyes. The processor 1800 may identify thebrightness of the IR light received through the image sensor of the IRcamera including the plurality of photodiodes, and may identify at leastone pixel corresponding to bright IR light equal to or greater than acertain reference among the pixels of the image captured by the IRcamera, thereby identifying the position of the glint feature point ofthe eyes. For example, the processor 1800 may identify the pixelcorresponding to the brightest IR light among the pixels of the imagecaptured by the IR camera, thereby identifying the position of the glintfeature point of the eyes. For example, the positions of the pixels inthe image captured by the IR camera may be identified through thecoordinate system of the IR camera, and the position of the glintfeature point of the eyes may have the coordinate value in thecoordinate system of the IR camera, as a position value of the pixelcorresponding to the glint feature point.

Alternatively, for example, when the light receiver 1520 is an IRdetector, the processor 1800 may calculate the coordinate value of thepupil feature point and the coordinate value of the glint feature pointof the eyes in the coordinate system of the IR detector.

When the light emitter 1510 is an IR scanner, the processor 1800 maycontrol the IR scanner to sequentially irradiate a point light sourceora line light source to cover a region where the user's eyes arelocated, and sequentially receive the light reflected from the user'seyes through the IR detector in order to scan the region where theuser's eyes are located. In addition, the processor 1800 may analyze anarray of light sequentially received through the IR detector, therebyidentifying the pupil feature point and the glint feature point of theeyes.

The coordinate system of the IR detector may be used to indicate theposition of the pupil feature point of the pupil and the position of theglint feature point of the eyes, and, for example, coordinate valuescorresponding to the lights in the array of lights sequentially receivedthrough the IR detector on the coordinate system of the IR detector maybe previously set. For example, irradiation directions and irradiationtimes of lights emitted from the IR scanner may be determined accordingto an operation setting value of the IR scanner, and a light array maybe formed from the lights emitted from the IR scanner. For example,based on the irradiation direction and irradiation time of the lightsemitted from the IR scanner, and reception time of the lights receivedfrom the IR detector, coordinate values corresponding to the lights inthe light array on the coordinate system of the IR detector may beidentified. In addition, based on the property (e.g., brightness) of thelights in the array of lights sequentially received through the IRdetector, light corresponding to the feature point of the eye andcoordinate values of the light may be identified.

For example, the processor 1800 may identify lights having a brightnessequal to or less than a certain value in the received light array,thereby identifying the position of the pupil feature point based oncoordinate values corresponding to the identified lights on thecoordinate system of the IR detector.

For example, the processor 1800 may identify light having a brightnessequal to or greater than a certain value in the received light array,thereby identifying a coordinate value corresponding to the identifiedlight on the coordinate system of the IR detector as the coordinatevalue of the glint feature point of the eyes.

Also, for example, when the light receiver 1520 is an IR detector, thecoordinate value of the pupil feature point and the coordinate value ofthe glint feature point of the eyes may be values calibrated byreflecting the degree of bias of the support 190 of the AR device 1000,which will be described below. In this case, the processor 1800 maycalculate the coordinate value of the pupil feature point and thecoordinate value of the glint feature point of the eyes calibrated byreflecting the degree of bias of the temple 191 of the AR device 1000and/or the degree of bias of the nose support 192. The calibratedcoordinate values may be input to a mapping function which will bedescribed below.

The processor 1800 may execute the pattern detection code 1740 stored inthe storage 1700, thereby detecting a pattern of the light reflector1400. The light reflector 1400 may be coated on one surface of thewaveguide 170 of the AR device 1000 to have a certain pattern. Theprocessor 1800 may receive the IR light reflected by the user's eyes andreflected by the light reflector 1400 through the light receiver 1520,and identify a shape of the pattern based on the received IR light. Thepattern formed on the light reflector 1400 may include, for example, adot pattern, a line pattern, a grid pattern, a 2D marker, etc., but thedisclosure is not limited thereto. An example of the pattern formed onthe light reflector 1400 and identified by the pattern detection code1740 will be described in more detail with reference to FIGS. 7A to 7D.When the temple 191 is biased with respect to the frame 110, the patternidentified by the pattern detection code 1740 may have a deformed shape.

For example, when the light receiver 1520 is an IR camera, the IR cameramay capture the user's eyes based on the IR light reflected by the lightreflector 1400, and the processor 1800 may identify a pattern within animage obtained by capturing the user's eyes from the image.

For example, when the light receiver 1520 is an IR detector, the IRdetector may sequentially receive IR lights reflected by the lightreflector 1400, and the processor 1800 may identify a part related tothe pattern of the light reflector 1400 in an array of the sequentiallyreceived IR lights.

The processor 1800 may execute the bias determination code 1750 storedin the storage 1700, thereby determining a degree to which the support190 of the AR device 1000 is biased with respect to the frame 110. Thesupport 190 may include, for example, the temple 191 and the nosesupport 192. When the user wears the AR device 1000, the temple 191 maybe widened or narrowed according to the size of the user's head andface. At this time, when the IR light is received after the support 190is biased with respect to the frame 110, the bias determination code1750 may identify a pattern having a deformed shape from the received IRlight. Also, the bias determination code 1750 may compare the patternhaving the deformed shape with a non-deformed pattern, therebyestimating the degree of bias of the support 190. For example, the biasdetermination code 1750 may compare the pattern having the deformedshape with the non-deformed pattern to identify degree of deformation ofthe pattern, and may determine degree of bias of the support 190 basedon the degree of deformation of the pattern. Alternatively, the biasdetermination code 1750 may analyze only the pattern having the deformedshape without comparing the pattern having the deformed shape with thenon-deformed pattern, thereby estimating the degree of bias of thesupport 190. For example, when the pattern is a dot pattern, the biasdetermination code 1750 may identify a difference between intervalsbetween points in the pattern having the deformed shape, therebyestimating the degree of bias of the support 190.

For example, the degree of bias of the temple 191 may be expressed as abias angle indicating a difference between a default angle of the temple191 with respect to the frame 110 and an angle of the biased temple 191with respect to the frame 110, but the disclosure is not limitedthereto. Also, for example, the degree of bias of the nose support 192may be expressed as a bias angle indicating a difference between adefault angle of the nose support 192 with respect to the frame 110 andan angle of the biased nose support 192 with respect to the frame 110,but the disclosure is not limited thereto.

In the above, it has been described that the pattern detection code 1740detects the deformation of the pattern formed on the light reflector1400, and the bias determination code 1750 analyzes the pattern havingthe deformed shape to estimate the degree of bias of the support 190,but the disclosure is not limited thereto.

For example, a certain pattern may be formed on a part of the AR device1000 that may reflect light emitted from the light emitter 1510 todirect the reflected light toward the light receiver 1520. For example,the pattern may be formed on a partial region of the frame 110 of the ARdevice 1000 directed toward the light emitter 1510 and the lightreceiver 1520. For example, the pattern may be formed by forming acertain curve in a partial region of the frame 110. Alternatively, forexample, the pattern may be formed by forming a material capable ofreflecting light on a partial region of the frame 110. In addition, thepattern may be attached to or coated on other elements included in theglasses type AR device 1000, for example, a vision correcting lens forvision correction or a cover glass installed to protect a waveguide.

Alternatively, for example, the pattern may be formed on a partialregion of the nose support 192 of the AR device 1000 directed toward thelight emitter 1510 and the light receiver 1520. For example, the patternmay be formed by forming a certain curve in a partial region of the nosesupport 192. Alternatively, for example, the pattern may be formed byforming a material capable of reflecting light on a partial region ofthe nose support 192. In this case, the light emitter 1510 and the lightreceiver 1520 are preferably located in the temple 191 of the AR device1000, but the disclosure is not limited thereto.

Alternatively, for example, the pattern may be formed in a partialregion of the temple 191 of the AR device 1000 directed toward the lightemitter 1510 and the light receiver 1520. For example, the pattern maybe formed by forming a certain curve in a partial region of the temple191. Alternatively, for example, the pattern may be formed by forming amaterial capable of reflecting light on a partial region of the temple191. In this case, the light emitter 1510 and the light receiver 1520are preferably located on the nose support 192 of the AR device 1000,but the disclosure is not limited thereto.

The processor 1800 may execute the pupil position detection code 1760stored in the storage 1700, thereby detecting a pupil position of theuser's eyes. The pupil position detection code 1760 may identify thepupil position of the user's eyes based on the IR light reflected fromthe light reflector 1400.

For example, when the light receiver 1520 is an IR camera, the pupilposition detection code 1760 may identify the pupil position of theuser's eyes within an image captured by the IR camera from the image.Alternatively, for example, when the light receiver 1520 is an IRdetector, the pupil position detection code 1760 may analyze the IRlight sequentially obtained by the IR detector, thereby calculating thepupil position of the user's eyes.

The pupil position detection code 1760 may identify the pupil centralpoint of the user's eyes, thereby identifying the pupil position of theuser's eyes.

The processor 1800 may execute the gaze determination code 1770 storedin the storage 1700, thereby obtaining information about the user'sgaze. The processor 1800 may execute the gaze determination code 1770,thereby calculating a position of the center of the user's eyes. Thecenter of the user's eyes may be the center of user's eyeballs. Theprocessor 1800 may calculate the position of the center of the user'seyes, based on the pupil position of the user's eyes obtained by thepupil position detection code 1760 and the degree of bias of the support190 obtained by the bias determination code 1750. For example, theprocessor 1800 may calculate the position of the center of the user'seyes so that a value calculated based on a matrix for calibrating thedegree of bias of the support 190, a value indicating the position ofthe center of the user's eyes and a bias of axis of the image obtainedby capturing the user's eyes can be a value of the pupil position of theuser's eyes obtained by the pupil position detection code 1760. Forexample, the center of the eye may be the center of the eyeball, and theposition of the center of the user's eyes may have a 3D coordinate valuein a coordinate system of a real space.

The processor 1800 may execute the gaze determination code 1770, therebycalculating a position of the gaze point of the user. In order tocalculate the position of the gaze point of the user, the processor 1800may previously generate a mapping function for calculating the positionof the gaze point from features of the user's eyes. The mapping functionis a function for calculating the position of the gaze point of the userin consideration of features of the user's eyes and bias information ofthe support 190, and may be generated during a calibration process ofthe calibration code 1780 which will be described below. For example,the position of the gaze point may have a 3D coordinate value in thecoordinate system in the real space, but the disclosure is not limitedthereto. For example, the position of the gaze point may have acoordinate value in the coordinate system of the waveguide 170, but thedisclosure is not limited thereto.

The processor 1800 may execute the gaze determination code 1770, therebycalibrating the features related to the user's gaze obtained from theeye feature detection code 1730 based on the degree of bias obtainedfrom the bias determination code 1750. Also, the processor 1800 mayapply the features related to the user's gaze calibrated based on thedegree of bias to the mapping function, thereby calculating the positionof the gaze point of the user. Also, a gaze direction of the user may bedetermined based on the position of the central point of the user's eyesand the gaze point of the user calculated by the gaze determination code1770. A method of obtaining the gaze direction of the user by using thegaze determination code 1770 will be described in more detail withreference to FIG. 11.

Alternatively, the processor 1800 may calculate the gaze point of theuser without using the above-described mapping function. For example,when the light receiver 1520 is an IR camera, the gaze determinationcode 1770 may calculate the gaze direction of the user's eyes from theimage obtained by capturing the user's eyes by using a certainalgorithm. In this case, the obtained gaze direction may be a vectorvalue indicating the gaze direction of the user's eyes in the cameracoordinate system. The algorithm used to obtain the gaze direction ofthe user's eyes may be an algorithm for fitting a 3D eye model. Thealgorithm for fitting the 3D eye model may be an algorithm for obtaininga vector value indicating the gaze direction of the user by comparing aneye image corresponding to a reference vector value indicating the gazedirection of the user with an image captured by an IR camera Inaddition, the gaze determination code 1770 may convert the vector valueindicating the gaze direction in the camera coordinate system into avector value indicating the gaze direction in the coordinate system ofthe waveguide170 by using the bias angle of the temple 191. Thereafter,the gaze determination code 1770 may calculate an intersection pointbetween a vector indicating the gaze direction in the coordinate systemof the waveguide 170 and the waveguide 170, thereby obtaining the gazepoint of the user.

The processor 1800 may execute the calibration code 1780 stored in thestorage 1700, thereby calibrating the mapping function based on the biasangle of the support 190. The processor 1800 may execute the calibrationcode 1780, thereby calibrating the mapping function to obtain the gazepoint of the user based on a previously set default bias angle andfeatures of eyes.

For example, when the light receiver 1520 is an IR camera, the processor1800 may display a target point for calibration through thewaveguide170, and capture the user's eyes looking at the target point byusing the IR camera. In addition, the processor 1800 may identify andanalyze a pattern within the image obtained by capturing the user'seyes, thereby obtaining a bias angle of the support 190. In addition,the processor 1800 may detect positions of the feature points related tothe user's eyes from the image obtained by capturing the user's eyes,and input the positions of the feature points of the user's eyes and thebias angle of the support 190 into the mapping function. The processor1800 may calibrate the mapping function so that a position value of thetarget point may be output from the mapping function to which thepositions of the feature points of the user's eyes and the bias angle ofthe support 190 are input.

For example, when the light receiver 1520 is an IR detector, theprocessor 1800 may display the target point for calibration through thewaveguide170 and control the IR scanner to emit IR light for scanningthe user's eyes looking at the target point. In addition, the processor1800 may receive and analyze IR light reflected from the user's eyesthrough the IR detector, identify a pattern of the light reflector 1400,and estimate the bias angle of the temple 191. The processor 1800 mayanalyze the IR light based on the bias angle of the temple 191, therebyidentifying positions of the calibrated feature points of eyes. Forexample, when the processor 1800 estimates the positions of the featurepoints of eyes by using the IR scanner and the IR detector, becauseresults of estimating the positions of the feature points of eyes areaffected by an operating angle of the IR scanner, the positions of thefeature points of eyes may be calibrated based on the bias of thesupport 190 by calculating a value obtained by subtracting the biasangle of the support 190 from the operating angle of the IR scanner.

In addition, the processor 1800 may input the positions of thecalibrated feature points of the eyes into the mapping function, andcalibrate the mapping function so that the position value of the targetpoint may be output from the mapping function in which the calibratedpositions of the feature points of the eyes are input.

FIG. 4 is a diagram illustrating an example of operations of the lightemitter 1510 and the light receiver 1520 of the AR device 1000 accordingto an example embodiment of the disclosure.

Referring to FIG. 4, the light emitter 1510 may emit IR light toward thelight reflector 1400, and the emitted IR light may be reflected by thelight reflector 1400 and directed toward user's eyes. In addition, theIR light directed toward the user's eyes may be reflected back by theuser's eyes and directed toward the light reflector 1400, and the IRlight reflected by the user's eyes may be reflected back by the lightreflector 1400 and directed toward the light receiver 1520. Also, thelight receiver 1520 may receive IR light reflected from the user's eyesby the light reflector 1400 and directed toward the light receiver 1520.

In FIG. 4, for convenience of explanation, it has been described thatthe AR device 1000 which is a glasses type display device emits IR lighttoward the user's left eye and receives reflected IR light from theuser's left eye, but the disclosure is not limited thereto. The ARdevice 1000 may emit IR light toward the user's right eye and receivethe reflected IR light from the user's right eye in the same manner asshown in FIG. 4.

FIG. 5A is a diagram illustrating an example of the light emitter 1510that emits planar light according to an example embodiment of thedisclosure.

Referring to FIG. 5A, the light emitter 1510 of FIG. 2 may be an IR LED,and the light receiver 1520 of FIG. 2 may be an IR camera. In this case,the light emitter 1510 may emit IR light of the planar light toward thelight reflector 1400, and the emitted IR light may be reflected by thelight reflector 1400 and directed toward a user's eye. For example, inorder to reflect the light emitted from the IR LED by using the lightreflector 1400 and irradiate the light to the region including theuser's eye, the processor 1800 may control an irradiation direction ofthe IR light emitted from the IR LED, and apply power to the IR LED,thereby controlling emission of the IR light from the IR LED. Inaddition, the IR light of the planar light reflected by the lightreflector 1400 may cover the entire user's eye. In this case, the lightreceiver 1520 may be an IR camera, and the IR camera may receive the IRlight reflected by the user's eye, thereby capturing the user's eye.

For example, coordinate values corresponding to pixels of an imagecaptured by the IR camera on the coordinate system of the IR camera maybe previously set. In addition, the processor 1800 may identify acoordinate value corresponding to a feature point of the eye based on aproperty (e.g., brightness) of IR light received through the IR camera.For example, the processor 1800 may identify the brightness of IR lightreceived through an image sensor of the IR camera including a pluralityof photodiodes, and identify at least one pixel that receives IR lightindicating the pupil among the pixels of the image captured by the IRcamera, thereby identifying a coordinate value 51 corresponding to thepupil central point on the IR coordinate system. In this case, the IRlight representing the pupil may be an IR light having a brightnesslower than a certain value, but the disclosure is not limited thereto.

Also, for example, the processor 1800 may identify the brightness of theIR light received through the image sensor of the IR camera includingthe plurality of photodiodes, and may identify at least one pixelrepresenting the glint feature point of the eye among the pixels of theimage captured by the IR camera, thereby identifying a coordinate value52 corresponding to a glint feature point on the IR coordinate system.In this case, the IR light representing the glint feature point of theeye may be an IR light having a brightness greater than a certain value,but the disclosure is not limited thereto.

According to an example embodiment of the disclosure, the light emitter1510 may be an IR LED that emits blinking light, and in this case, thelight reflector 1400 may be an IR event camera. The IR event camera maybe an IR camera that is activated when a specific event occurs andautomatically captures a subject. The IR event camera may be activatedto automatically capture the user's eye, for example, when patterns ofblinking light are different.

FIG. 5B is a diagram illustrating an example of the light emitter 1510that emits point light according to an example embodiment of thedisclosure.

Referring to FIG. 5B, the light emitter 1510 of FIG. 2 may be an IRscanner, and the light receiver 1520 of FIG. 2 may be an IR detector. Inthis case, the light emitter 1510 may emit IR light of a point lighttoward the light reflector 1400, and the emitted IR light may bereflected by the light reflector 1400 and directed toward user's eye. Inthis case, the light emitter 1510 may sequentially emit IR lights ofpoint light toward the light reflector 1400 while changing an emissiondirection to a vertical direction or a horizontal direction, and thesequentially emitted IR lights of point light may be reflected by thelight reflector 1400 to cover the entire user's eye. For example, inorder to reflect the IR lights of point light sequentially emitted fromthe IR scanner by using the light reflector 1400 and irradiate the IRlights of point light to a region including the user's eye, theprocessor 1800 may control irradiation directions of the IR lightsemitted from the IR scanner. In this case, the light emitter 1510 may bea 2-dimensional (2D) scanner and the light receiver 1520 may be at leastone photodiode.

For example, coordinate values corresponding to the IR lights in thearray of lights sequentially received through the IR detector on thecoordinate system of the IR detector may be previously set, and theprocessor 1800 may identify a coordinate value corresponding to afeature point of the eye based on properties (e.g., brightness) of theIR lights in the array of IR lights sequentially received through the IRdetector. For example, the processor 1800 may identify coordinate valuescorresponding to IR lights having a brightness equal to or less than acertain value in the array of the received IR lights on the coordinatesystem of the IR detector, thereby identifying a coordinate value 53corresponding to the pupil center point on the IR coordinate system.Also, for example, the processor 1800 may identify coordinate valuescorresponding to IR lights having a brightness equal to or less than acertain value in the array of the received IR lights on the coordinatesystem of the IR detector, thereby identifying a coordinate value 54corresponding to the glint feature point of the eye on the IR coordinatesystem.

FIG. 5C is a diagram illustrating an example of the light emitter 1510that emits line light according to an example embodiment of thedisclosure.

Referring to FIG. 5C, the light emitter 1510 of FIG. 2 may be an IRscanner, and the light receiver 1520 of FIG. 2 may be an IR detector. Inthis case, the light emitter 1510 may emit IR light of line light towardthe light reflector 1400, and the emitted IR light may be reflected bythe light reflector 1400 and directed toward user's eye. For example, inorder to reflect the IR lights of line light sequentially emitted fromthe IR scanner by using the light reflector 1400 and irradiate the IRlights of point light to a region including the user's eye, theprocessor 1800 may control irradiation directions of the IR lightsemitted from the IR scanner. In this case, the light emitter 1510 maysequentially emit IR lights of line light toward the light reflector1400 while changing an emission direction, and the sequentially emittedIR lights of line light may be reflected by the light reflector 1400 tocover the entire user's eye. In this case, the light emitter 1510 may bea 1-dimensional (1D) scanner and the light receiver 1520 may be aphotodiode array including a plurality of photodiodes. When the lightreceiver 1520 is the photodiode array, it is more preferable that thelight receiver 1520 is provided in the temple 191. For example, whenhorizontal line light is emitted from the light emitter 1510, the lightemitter 1510 may change the emission direction to a vertical directionto cover the region including the user's eye, and, when a vertical linelight is emitted from the light emitter 1510, the light emitter 1510 maychange the emission direction to the horizontal direction.

For example, coordinate values corresponding to the IR lights in thearray of lights sequentially received through the IR detector on thecoordinate system of the IR detector may be previously set, and theprocessor 1800 may identify a coordinate value corresponding to afeature point of the eye based on properties (e.g., brightness) of theIR lights in the array of IR lights sequentially received through the IRdetector. For example, the processor 1800 may identify line lights of IRlight having a brightness greater than or equal to a certain value, andidentify a photodiode corresponding to IR light having a brightness lessthan or equal to the certain value among a plurality of photodiodes thathave received the line lights according to the respective line lights,thereby identifying a coordinate value 55 corresponding to the pupilcenter point on the IR coordinate system. In addition, for example, theprocessor 1800 may identify line lights of IR light having a brightnessgreater than or equal to a certain value, and identify a photodiodecorresponding to IR light having a brightness less than or equal to thecertain value among a plurality of photodiodes that have received theline lights according to the respective line lights, thereby identifyinga coordinate value 56 corresponding to the glint feature point of theeye on the IR coordinate system.

In FIGS. 5A to 5C, for convenience of description, an example in whichthe AR device 1000 which is the glasses type display device, emits IRlight toward user's one eye has been described, but the disclosure isnot limited thereto. The AR device 1000 may emit IR light toward theuser's other eye in the same manner as illustrated in FIGS. 5A to 5C.

FIG. 6A is a diagram illustrating an example in which the light emitter1510 and the light receiver 1520 are provided in the temple 191 of theAR device 1000 according to an example embodiment of the disclosure.

Referring to FIG. 6A, the AR device 1000 of FIG. 2 may be a glasses typedevice, and the light emitter 1510 and the light receiver 1520 may beprovided on the temple 191 of the AR device 1000. The light emitter 1510and the light receiver 1520 may be provided on an inner side part of thetemple 191 of the AR device 1000, which is a position between the temple191 and user's eyes. For example, the light emitter 1510 and the lightreceiver 1520 may be provided at positions spaced from a frame by about10 mm to 15 mm on an inner side of the temple 191 of the AR device 1000The light emitter 1510 and the light receiver 1520 may be provided toface the light reflector 1400 in the temple 191 of the AR device 1000.

FIG. 6B is a diagram illustrating an example in which the light emitter1510 and the light receiver 1520 are provided in the nose support 192 ofthe AR device 1000 according to an example embodiment of the disclosure.

Referring to FIG. 6B, the AR device 1000 of FIG. 2 may be a glasses typedevice, and the light emitter 1510 and the light receiver 1520 may beprovided on the nose support 192 of the AR device 1000. The lightemitter 1510 and the light receiver 1520 may be provided on an innerside part of the nose support 192 of the AR device 1000, which is aposition between the nose support 192 and user's eyes. The light emitter1510 and the light receiver 1520 may be provided to face the lightreflector 1400 in the nose support 192 of the AR device 1000.

FIG. 6C is a diagram illustrating an example in which the light emitter1510 and the light receiver 1520 are provided in the temple 191 and thenose support 192 of the AR device 1000 according to an exampleembodiment of the disclosure.

Referring to FIG. 6C, the AR device 1000 of FIG. 2 may be a glasses typedevice, the light emitter 1510 may be provided on the temple 191 of theAR device 1000, and the light receiver 1520 may be provided on the nosesupport 192 of the AR device 1000. The light emitter 1510 may beprovided on an inner side part of the temple 191 of the AR device 1000,which is a position between the temple 191 and user's eyes. The lightreceiver 1520 may be provided on an inner side part of the nose support192 of the AR device 1000, which is a position between the nose support192 and user's eyes. The light emitter 1510 and the light receiver 1520may be provided to face the light reflector 1400 in the AR device 1000.

FIG. 6D is a diagram illustrating an example in which the light emitter1510 and the light receiver 1520 are provided in the temple 191 and thenose support 192 of the AR device 1000 according to an exampleembodiment of the disclosure.

Referring to FIG. 6D, the AR device 1000 of FIG. 2 may be a glasses-typedevice, the light emitter 1510 may be provided on the nose support 192of the AR device 1000, and the light receiver 1520 may be provided onthe temple 191 of the AR device 1000. The light emitter 1510 may beprovided on an inner side part of the nose support 192 of the AR device1000, which is a position between the nose support 192 and user's eyes.The light receiver 1520 may be provided on an inner side part of thetemple 191 of the AR device 1000, which is a position between the temple191 and user's eyes. The light emitter 1510 and the light receiver 1520may be provided to face the light reflector 1400 in the AR device 1000.

In FIGS. 6A to 6D, it has been described that one light emitter 1510 andone light receiver 1520 are provided in the AR device 1000, but thedisclosure is not limited thereto. For example, a plurality of lightemitters 1510 may be provided in the AR device 1000. In this case, theplurality of light emitters 1510 may be provided on the temple 191, orthe plurality of light emitters 1510 may be provided on the nose support192. Alternatively, the plurality of light emitters 1510 may bedividedly provided on the temple 191 and the nose support 192.

Also, for example, the plurality of light receivers 1520 may be providedin the AR device 1000. In this case, the plurality of light receivers1520 may be provided on the temple 191, or the plurality of lightreceivers 1520 may be provided on the nose support 192. Alternatively,the plurality of light receivers 1520 may be dividedly provided on thetemple 191 and the nose support 192.

In FIGS. 6A to 6D, for convenience of description, it has been describedthat the light emitter 1510 and the light receiver 1520 are provided ina left eye part of the AR device 1000, which is the glasses type displaydevice, but the disclosure is not limited thereto. In the AR device1000, the light emitter 1510 and the light receiver 1520 may be providedin a right eye part of the AR device 1000 in the same manner asillustrated in FIGS. 6A to 6D.

FIG. 7A is a diagram illustrating an example of a dot pattern formed onthe light reflector 1400 of the AR device 1000 according to an exampleembodiment of the disclosure, FIG. 7B is a diagram illustrating anexample of a grid pattern formed on the light reflector 1400 of the ARdevice 1000 according to an example embodiment of the disclosure, FIG.7C is a diagram illustrating an example of a pattern in the form of a 2Dmarker according to an example embodiment of the disclosure, and FIG. 7Dis a diagram illustrating an example of the light reflector 1400 thatcovers a part of the waveguide 170 according to an example embodiment ofthe disclosure.

Referring to FIG. 7A, the dot-shaped pattern may be formed on the lightreflector 1400 of the AR device 1000 of FIG. 2. Referring to FIG. 7B,the grid-shaped pattern may be formed on the light reflector 1400 of theAR device 1000 of FIG. 2. According to an example embodiment, IR lightmay not be reflected from a part where the pattern is formed. At thistime, because the dot-shaped pattern or the grid-shaped pattern is fordetecting bias of a support or a temple of an AR device, it ispreferable to have a regular shape.

Referring to FIG. 7C, the pattern formed on a part of the lightreflector 1400 of the AR device 1000 of FIG. 2 at which the user's gazeis less frequently directed. The formed pattern may be, for example, apattern in the form of the 2D marker, but the disclosure is not limitedthereto. The pattern may be formed on, for example, a part of the lightreflector 1400 that does not interfere with capturing or scanning theuser's eyes.

Referring to FIG. 7D, the light reflector 1400 may be formed on a partof the waveguide 170 of the AR device 1000 of FIG. 2. For example, thelight reflector 1400 may not be located in a part of the waveguide 170which has little relation to reflection of IR light.

For example, when the light receiver 1520 is an IR camera, the IR cameramay capture the user's eyes based on the IR light reflected by the lightreflector 1400, and the IR light may not be reflected from the partwhere the pattern is formed. For example, a part of an image obtained bycapturing the user's eyes in which the IR light is not reflected may bein black, and the processor 1800 may identify the black part in theimage obtained by capturing the user's eyes, thereby identifying thepattern in the image.

For example, when the light receiver 1520 is an IR detector, the IRdetector may sequentially receive IR lights reflected by the lightreflector 1400, and the IR light may not be reflected from the partwhere the pattern is formed. For example, the processor 1800 mayidentify a part of an IR light array formed by the sequentially receivedIR lights, from which the IR light is not reflected, thereby identifyingthe pattern of the light reflector 1400.

FIG. 8A is a diagram illustrating a light emission angle and a patternbefore the temple 191 of the AR device 1000 is biased according to anexample embodiment of the disclosure, and FIG. 8B is a diagramillustrating a light emission angle and a pattern after the temple 191of the AR device 1000 is biased according to an example embodiment ofthe disclosure.

Referring to FIG. 8A, in the glasses type AR device 1000 as shown inFIG. 2, before the temple 191 is biased, the light emitter 1510 may emitIR light toward the light reflector 1400 on which a point pattern isformed. According to an example embodiment, the light emitter 1510 mayemit IR light toward the light reflector 1400 at a first light emissionangle 80, and a first pattern 82 may be identified by the AR device 1000based on the IR light received by the light receiver 1520.

In addition, referring to FIG. 8B, in the glasses type AR device 1000 asshown in FIG. 2, after the temple 191 is biased, the light emitter 1510may emit IR light toward the light reflector 1400 at a second lightemission angle 90, and a second pattern 92 may be identified by the ARdevice 1000 based on the IR light received by the light receiver 1520.

As shown in FIGS. 8A and 8B, the first pattern 82 and the second pattern92 may include dots with different spaces from each other, and the ARdevice 1000 may compare the first pattern 82 and the second pattern 92,thereby identifying degree of bias of the temple 191. For example, abias angle with respect to the first pattern 82 may be set to ‘0’, and abias angle with respect to the second pattern 92 may be calculated basedon differences in positions between points in the first pattern 82 andpoints in the second pattern 92. For example, the bias angle withrespect to the second pattern 92 may be a difference value between thefirst light emission angle 80 and the second light emission angle 90.For example, when a pattern corresponding to the first light emissionangle 80 is the first pattern 82 and a pattern corresponding to thefirst light emission angle 80 is the second pattern 92, the processor1800 may compare spaces of the dots in the first pattern 82 with thespaces of the dots in the second pattern 92, thereby identifyingdifference values of the spaces of the dots in the second pattern 92with respect to the spaces of the dots in the first pattern 82, and mayinput the identified difference values into at least one function forcalculating degree of bias of the temple 191, thereby calculating thedegree of bias of the temple 191 indicating a difference between thefirst light emission angle 80 corresponding to the first pattern 82 andthe second light emission angle 90 corresponding to the second pattern92.

In addition, for example, when a pattern corresponding to the firstlight emission angle 80 is the first pattern 82 and a patterncorresponding to the first light emission angle 80 is the second pattern92, the processor 1800 may compare positions of the dots in the firstpattern 82 with the positions of the dots in the second pattern 92,thereby identifying difference values of the positions of the dots inthe second pattern 92 with respect to the positions of the dots in thefirst pattern 82, and may input the identified difference values into atleast one function for calculating degree of bias of the temple 191,thereby calculating the degree of bias of the temple 191 indicating adifference between the first light emission angle 80 corresponding tothe first pattern 82 and the second light emission angle 90corresponding to the second pattern 92.

According to another example embodiment, the processor 1800 may identifythe degree of bias of the temple 191 based on the second pattern 92without comparing the first pattern 82 and the second pattern 92. Inthis case, the processor 1800 may identify the difference values of thespaces of the dots in the second pattern 92, and input the identifieddifference values into at least one function for calculating the degreeof bias of the temple 191, thereby calculating the degree of bias of thetemple 191.

FIG. 9 is a diagram illustrating an example of a pattern identified froman array of light received through the light receiver 1520 when a lightemitter of the AR device 1000 is an IR scanner 1520-1 or 1520-2according to an example embodiment of the disclosure.

Referring to FIG. 9, the IR scanner 1520-1 may represent an IR scannerbefore the support 190 of the glasses type AR device 1000 as shown inFIG. 2 is biased. The IR scanner 1520-1 may sequentially emit IR lightof point lint toward a region where user's eyes are located by using thelight reflector 1400, and the light receiver 1520 may receive a firstlight array 90.

In addition, the IR scanner 1520-2 may represent the IR scanner afterthe support 190 of the AR device 1000 is biased. The IR scanner 1520-2may sequentially emit IR light of point lint toward the region whereuser's eyes are located by using the light reflector 1400, and the lightreceiver 1520 may receive a second light array 92.

In the first light array 90 and the second light array 92, positions andspaces of light signals corresponding to dot patterns of the lightreflector 1400 may be different from each other, and the AR device 1000may compare a part corresponding to the dot pattern of the lightreflector 1400 in the first light array 90 and a part corresponding tothe dot pattern of the light reflector 1400 in the second light array92. Also, the AR device 1000 may identify degree of bias of the support190 of the AR device 1000 based on results of comparison.

For example, because IR light emitted toward the dot pattern is notreflected by the light reflector 1400, light signal may not be receivedin parts 90-1, 90-2, 90-3, and 90-4 of the first light array 90corresponding to dot patterns and parts 92-1, 92-2, 92-3, and 92-4 ofthe second light array 92 corresponding to dot patterns. The processor1800 may identify parts of the first light array 90 from which no lightsignal is received, thereby identifying the parts 90-1, 90-2, 90-3, and90-4 of the first light array 90 corresponding to the dot patterns, andidentifying coordinate values 90-5, 90-6, 90-7, and 90-8 on a coordinatesystem of the IR scanner 1520-1 respectively indicating the parts 90-1,90-2, 90-3, and 90-4 corresponding to the dot patterns. Also, forexample, the processor 1800 may identify parts of the second light array92 from which no light signal is received, thereby identifying the parts92-1, 92-2, 92-3, and 92-4of the second light array 92 corresponding tothe dot patterns, and identifying coordinate values 92-5, 92-6, 92-7,and 92-8 on a coordinate system of the IR scanner 1520-2 respectivelyindicating the parts 92-1, 92-2, 92-3, and 92-4 corresponding to the dotpatterns.

The processor 1800 may compare the coordinate values 90-5, 90-6, 90-7,and 90-8 on the coordinate system of the IR scanner 1520-1 respectivelyindicating the parts 90-1, 90-2, 90-3, and 90-4 of the first light array90 corresponding to the dot patterns with the coordinate values 92-5,92-6, 92-7, and 92-8 on the coordinate system of the IR scanner 1520-2respectively indicating the parts 92-1, 92-2, 92-3, and 92-4 of thesecond light array 92 corresponding to the dot patterns, therebyidentifying differences between the coordinate values 90-5, 90-6, 90-7,and 90-8 and the coordinate values 92-5, 92-6, 92-7, and 92-8. Forexample, the processor 1800 may calculate the degree of bias of thesupport 190 of the AR device 1000, based on a difference value betweenthe coordinate value 90-5 and the coordinate value 92-5, a differencevalue between the coordinate value 90-6 and the coordinate value 92-6, adifference value between the coordinate value 90-7 and the coordinatevalue 92-7, and a difference value between the coordinate value 90-8 andthe coordinate value 92-8.

In FIG. 9, for convenience of description, the IR scanner 1520-1 beforethe support 190 is biased and the IR scanner 1520-2 after the support190 of the AR device 1000 is biased are illustrated separately in FIG.9, in order to distinguish the IR scanner 1520-1 before the support 190of the AR device 1000 is biased from the IR scanner 1520-2 after thesupport 190 of the AR device 1000 is biased. However, the IR scanner1520-1 before the support 190 is biased and the IR scanner 1520-2 afterthe support 190 of the AR device 1000 may be a same IR scanner installedin the AR device 1000.

In addition, in FIG. 9, for convenience of description, it has beendescribed that one first light array 90 is used to identify the dotpattern using the IR scanner 1520-1 before the support 190 is biased,and one second light array 92 is used to identify the dot pattern usingthe IR scanner 1520-2 after the support 190 of the AR device 1000 isbiased, but the disclosure is not limited thereto. A plurality of lightarrays for covering the dot pattern may be used to identify the dotpattern using the IR scanner 1520-1 before the support 190 is biased,and a plurality of light arrays for covering the dot pattern may be usedto identify the dot pattern using the IR scanner 1520-2 after thesupport 190 is biased.

FIG. 10 is a diagram illustrating an example of an eye featureidentified from an array of light received through the light receiver1520 when a light emitter of the AR device 1000 is the IR scanner 1520-1or 1520-2 according to an example embodiment of the disclosure.

Referring to FIG. 10, the IR scanner 1520-1 before the support 190 ofthe glasses type AR device 1000 as shown in FIG. 2 is biased maysequentially emit IR light of point lint toward a region where user'seyes are located by using the light reflector 1400, and the lightreceiver 1520 may receive a third light array 100.

In addition, after the support 190 of the AR device 1000 is biased, theIR scanner 1520-2 may sequentially emit IR light of point lint towardthe region where user's eyes are located by using the light reflector1400, and the light receiver 1520 may receive a fourth light array 102.

In the third light array 100 and the fourth light array 102, positionsand spaces of light signals corresponding to feature points of eyes maybe different from each other. The AR device 1000 may calibrate positionsof parts corresponding to the feature points of the eyes from the fourthlight array 102 in consideration of a bias angle of the support 190.

For example, the processor 1800 may identify parts 102-1 in the fourthlight array 102 corresponding to the glint feature points of the eyes,based on brightness of lights in the fourth light array 102, andidentify coordinate values 102-2 on the coordinate system of the IRscanner indicating the parts 102-1 corresponding to the glint featurepoints of the eyes. Thereafter, the processor 1800 may calibrate thecoordinate values 102-2 indicating the glint feature points of the eyesby using the bias angle calculated in FIG. 9. For example, the processor1800 may multiply the coordinate values 102-2 representing the glintfeature points of the eyes by a compensation matrix 18 of FIG. 11 whichwill be described below, thereby obtaining calibrated coordinate values.

In FIG. 10, for convenience of description, the IR scanner 1520-1 beforethe support 190 is biased and the IR scanner 1520-2 after the support190 of the AR device 1000 is biased are illustrated separately in FIG.10, in order to distinguish the IR scanner 1520-1 before the support 190of the AR device 1000 is biased from the IR scanner 1520-2 after thesupport 190 of the AR device 1000 is biased. However, the IR scanner1520-1 before the support 190 is biased and the IR scanner 1520-2 afterthe support 190 of the AR device 1000 are the same IR scanners installedin the AR device 1000.

In addition, in FIG. 10, for convenience of description, it has beendescribed that third light array 100 is used to identify feature pointsof eyes using the IR scanner 1520-1 before the support 190 is biased,and one fourth light array 102 is used to identify feature points ofeyes using the IR scanner 1520-2 after the support 190 of the AR device1000 is biased, but the disclosure is not limited thereto. A pluralityof light arrays for covering user's eyes may be used to identify featurepoints of eyes using the IR scanner 1520-1 before the support 190 isbiased, and a plurality of light arrays for covering user's eyes may beused to identify feature points of eyes using the IR scanner 1520-2after the support 190 is biased.

FIG. 11 is a diagram illustrating examples of functions used by the ARdevice 1000 to calculate a center of an eyeball and calculate a gazepoint 16 of a user according to an example embodiment of the disclosure.

Equation 11 represents a relationship between a coordinate value 12 of apupil center of the eye in a coordinate system of an IR camera and acoordinate value 13 in real space representing the center of theeyeball. In an example embodiment of the disclosure, the coordinatevalue 12 of the coordinate system of the IR camera may have a 2Dcoordinate value, and the coordinate value 13 of the real space may havea 2D coordinate value or a 3D coordinate value.

For example, when the coordinate value 13 representing the center of theeyeball is multiplied by a camera rotation matrix 20 and a scale factorand a value representing a bias in an image is added, the coordinatevalue 12 of the pupil center of the eye may be calculated. The camerarotation matrix 20 is a matrix that converts a coordinate value in realspace into a coordinate value of a camera coordinate system inconsideration of a position in which a camera is provided. Thecoordinate value 13 representing the center of the eyeball may beconverted into the coordinate value of the coordinate system of the IRcamera by multiplying the coordinate value 13 representing the center ofthe eyeball by the camera rotation matrix 20, and the size of theconverted coordinate value may be normalized by multiplying theconverted coordinate value by a scale factor. In addition, by adding avalue indicating the bias in the image to the normalized size of thecoordinate value, a normalized 2D coordinate value may be corrected byreflecting the bias in the image so that the coordinate value 12 of thepupil center of the eye may be calculated. In addition, the valuerepresenting the bias in the image may be used to arrange a position ofthe eye in an image captured by the IR camera in a reference position.For example, the value representing the bias in the image may be a valuefor moving a center point of the eye in the captured image to a centerpoint of the captured image, based on a difference between the centerpoint of the image captured by the IR camera and the center point of theeye in the captured image.

In an example embodiment of the disclosure, the camera rotation matrix20, the scale factor and the value representing the bias in the imagemay be determined in consideration of, for example, a position in whichan IR LED is provided, a position in which the IR camera is provided, acapturing direction of the IR camera, an angle of view of the IR camera,the image captured by the IR camera, information related to the humaneye (e.g., eyeball size, pupil size, etc.), previously captured eyeimages, etc., and may be previously set in the AR device 1000 duringmanufacturing of the AR device 1000.

In an example embodiment of the disclosure, the AR device 1000 may inputthe coordinate value 12 of the pupil center of the eye into Equation 11,thereby obtaining the coordinate value 13 representing the center of theeyeball.

Equation 15 may represent a relationship between values 17 representingfeature points of the eye and the gaze point 16 of the user. Forexample, the feature points calibrated by reflecting a bias of thesupport 190 may be obtained by multiplying the values 17 representingthe feature points of the eye by a compensation matrix 18. In addition,a value 19 that is output by inputting a value obtained by multiplyingthe values 17 representing the feature points of the eye by thecompensation matrix 18 into a mapping function F may be a coordinatevalue 16 representing the gaze point of the user. The compensationmatrix 18 may be a matrix for compensating for degree of a bias of thesupport 190. The compensation matrix 18 may be determined throughcomparison between images captured from the IR camera in a state inwhich the support 190 of the AR device 1000 of FIG. 2 is not biased andimages captured by the IR camera in a state in which the support 190 ofthe AR device 1000 is biased, and the compensation matrix 18 may bepreviously set in the AR device 1000 during manufacturing of the ARdevice 1000. For example, the compensation matrix 18 may be determinedso that the images captured from the IR camera in a state in which thesupport 190 is biased may be converted into the images captured by theIR camera in a state in which the support 190 is not biased, inconsideration of the degree of the bias of the support 190. However, theexample in which the compensation matrix 18 is determined is not limitedthereto.

For example, the compensation matrix 18 may be determined throughcomparison between information obtained from the images captured fromthe IR camera in a state in which the support 190 is not biased andinformation obtained from the images captured by the IR camera in astate in which the support 190 of the AR device 1000 is biased. Dataobtained from an image may include, for example, a size of the image, aposition of the eye in the image, a position of the pupil in the image,a position of a pattern in the image, etc., but the disclosure is notlimited thereto.

Alternatively, for example, the compensation matrix 18 may be determinedby comparing the images captured from the IR camera in a state in whichthe support 190 is not biased and the information obtained from theimages captured by the IR camera in a state in which the support 190 ofthe AR device 1000 is biased. Alternatively, for example, thecompensation matrix 18 may be determined by comparing the informationobtained from the images captured from the IR camera in a state in whichthe support 190 is not biased and the images captured by the IR camerain a state in which the support 190 of the AR device 1000 is biased.

In addition, the mapping function F, which is a function for calculatingthe gaze point of the user from the feature points of the eye, may bedetermined such that the gaze point of the user is calculated from thereward matrix 18 and the feature points of the eye, and may bepreviously set in the AR device 1000 during manufacturing of the ARdevice 1000.

The AR device 1000 may obtain the coordinate value 16 representing thegaze point of the user from positions of the feature points of the eyeusing Equation 15.

FIG. 12 is a flowchart of a method, performed by the AR device 1000 ofFIGS. 2 and 3, of detecting a user's gaze according to an exampleembodiment of the disclosure.

In operation S1200, the AR device 1000 may emit IR light toward thelight reflector 1400 through the light emitter 1510 installed on thesupport 190 extending from the frame 110 of the AR device 1000. The ARdevice 1000 may emit the IR light toward at least a partial region ofthe light reflector 1400 so that the IR light reflected by the lightreflector 1400 may cover user's eyes.

For example, when the light receiver 1520 is an IR camera, the lightemitter 1510 may be an IR LED, and the AR device 1000 may control the IRLED so that the IR light emitted from the IR LED may be reflected by thelight reflector 1400 and may cover the user's eyes, in order for the IRcamera to capture the user's eyes. Alternatively, for example, when thelight receiver 1520 is an IR detector, the light emitter 1510 may be anIR scanner, and the AR device 1000 may control the IR scanner to scanthe user's eyes by reflecting the IR light emitted from the IR scannerby using the light reflector 1400, so that the IR detector may detectthe user's eyes.

In operation S1205, the AR device 1000 may receive the IR lightreflected by the user's eye and reflected back by the light reflector1400, through the light receiver 1520 installed on the support 190extending from the frame 110 of the AR device 1000.

For example, when the light emitter 1510 is an IR LED, the lightreceiver 1520 may be an IR camera, and the AR device 1000 may controlthe IR camera to capture the user's eyes through the light reflected bythe light reflector 1400 from the user's eyes. Alternatively, forexample, when the light emitter 1510 is an IR scanner, the lightreceiver 1520 may be an IR detector, and the AR device 1000 may controlthe IR detector to detect the IR light reflected by the user's eye andreflected back by the light reflector 1400, so that the IR detector maydetect the user's eyes.

In operation S1210, the AR device 1000 may detect previously setfeatures related to the gaze of the user's eyes based on the received IRlight. For example, the AR device 1000 may detect a position of a pupilfeature point of the user's eyes and a position of a glint feature pointof the eyes. The pupil feature point may be, for example, a pupilcentral point, and the glint feature point of the eyes may be a parthaving brightness greater than or equal to a certain value in a detectedeye region. The position of the pupil feature point and the position ofthe glint feature point of the eyes may be identified, for example, by acoordinate value indicating a position in a coordinate system of thelight receiver 1520. For example, the coordinate system of the lightreceiver 1520 may be a coordinate system of an IR camera or a coordinatesystem of the IR detector, and the coordinate value in the coordinatesystem of the light receiver 1520 may be a 2D coordinate value.

The AR device 1000 may detect previously set features related to thegaze of the eyes by analyzing the light received by the light receiver1520. For example, when the light receiver 1520 is an IR camera, the ARdevice 1000 may identify the position of the pupil feature point and theposition of the glint feature point of the eyes in an image captured bythe IR camera. Alternatively, for example, when the light receiver 1520is an IR detector, the AR device 1000 may analyze the IR light detectedby the IR detector, thereby identifying the position of the pupilfeature point and the position of the glint feature point of the eyes.

In addition, the AR device 1000 may analyze the light received by thelight receiver 1520, thereby obtaining a coordinate value indicating theposition of the pupil feature point and a coordinate value indicatingthe position of the glint feature point of the eyes. For example, whenthe light receiver 1520 is an IR camera, the AR device 1000 may obtainthe coordinate value of the pupil feature point and the coordinate valueof the glint feature point of the eyes from the coordinate system of theIR camera. For example, when the light receiver 1520 is an IR camera,the AR device 1000 may identify the position of the pupil central pointin an image captured by the IR camera. For example, the position of thepupil central point may have a coordinate value in the coordinate systemof the IR camera.

For example, the AR device 1000 may identify a position of the brightestpoint in the image captured by the IR camera, in order to identify theglint feature point of the eyes. The AR device 1000 may identify thebrightness of the IR light received through an image sensor of the IRcamera including a plurality of photodiodes, and may identify at leastone pixel corresponding to bright IR light equal to or greater than acertain reference among pixels of the image captured by the IR camera,thereby identifying the position of the glint feature point of the eyes.For example, the AR device 1000 may identify the pixel corresponding tothe brightest IR light among the pixels of the image captured by the IRcamera, thereby identifying the position of the glint feature point ofthe eyes. For example, the position of the glint feature point of theeyes may have a coordinate value in the coordinate system of the IRcamera.

Alternatively, for example, when the light receiver 1520 is an IRdetector, the AR device 1000 may calculate the coordinate value of thepupil feature point and the coordinate value of the glint feature pointof the eyes in the coordinate system of the IR detector.

When the light emitter 1510 is an IR scanner, the AR device 1000 maycontrol the IR scanner to sequentially irradiate a point light sourceora line light source to cover a region where the user's eyes arelocated, and sequentially receive the light reflected from the user'seyes through the IR detector in order to scan the region where theuser's eyes are located. Also, the AR device 1000 may analyze an arrayof light sequentially received through the IR detector, therebyidentifying the pupil feature point and the glint feature point of theeyes. For example, the AR device 1000 may identify light having abrightness equal to or greater than a certain value in the receivedlight array, thereby identifying a coordinate of the glint feature pointof the eyes. For example, the position of the glint feature point of theeyes may have a coordinate value in the coordinate system of the IRdetector.

In operation S1215, the AR device 1000 may detect a pattern of the lightreflector 1400 based on the received IR light. The light reflector 1400may be coated on one surface of the waveguide 170 of the AR device 1000to have a certain pattern. The AR device 1000 may receive the IR lightreflected by the user's eyes and reflected by the light reflector 1400through the light receiver 1520, and identify a shape of the patternbased on the received IR light. The pattern formed on the lightreflector 1400 may include, for example, a dot pattern, a line pattern,a grid pattern, a 2D marker, etc., but the disclosure is not limitedthereto. When the temple 191 is biased with respect to the frame 110,the pattern identified by the pattern detection code 1740 may have adeformed shape.

For example, when the light receiver 1520 is an IR camera, the IR cameramay capture the user's eyes based on the IR light reflected by the lightreflector 1400, and the AR device 1000 may identify a pattern within animage obtained by capturing the user's eyes from the image. For example,when the light receiver 1520 is an IR detector, the IR detector maysequentially receive IR lights reflected by the light reflector 1400,and the AR device 1000 may identify a part related to the pattern of thelight reflector 1400 in an array of the sequentially received IR lights.

In operation S1220, the AR device 1000 may determine a degree to whichthe support 190 of the AR device 1000 is biased with respect to theframe 110. When the IR light is received after the support 190 is biasedwith respect to the frame 110, the AR device 1000 may identify a patternhaving a deformed shape from the received IR light. Also, for example,as in FIGS. 8A and 8B, the AR device 1000 may compare the pattern havingthe deformed shape with a non-deformed pattern, thereby estimating thedegree of bias of the support 190. For example, the degree of bias ofthe temple 191 may be expressed as a bias angle indicating a differencebetween a default angle of the temple 191 with respect to the frame 110and an angle of the biased temple 191 with respect to the frame 110, butthe disclosure is not limited thereto. Also, for example, the degree ofbias of the nose support 192 may be expressed as a bias angle indicatinga difference between a default angle of the nose support 192 withrespect to the frame 110 and an angle of the biased nose support 192with respect to the frame 110, but the disclosure is not limitedthereto.

Also, for example, when the light receiver 1520 is an IR detector, thecoordinate value of the pupil feature point and the coordinate value ofthe glint feature point of the eyes may be values calibrated byreflecting the degree of bias of the support 190 of the AR device 1000.When the light receiver 1520 is an IR detector, for example, when thecoordinate values 102-2 corresponding to the IR lights 102-1corresponding to the feature points of the eyes in the light array 102of FIG. 10 are calculated, the degree of bias of the support 190 may bereflected. For example, the AR device 1000 may calibrate positions ofthe lights 102-1 corresponding to the feature points of the eyes in thelight array 102 among the lights in the light array 102 received by theIR detector, by reflecting the degree of bias of the support 190. The ARdevice 1000 may directly calculate the coordinate values 102corresponding to the feature points of the eyes in the coordinate systemof the IR detector, based on the calibrated positions of the lights102-1 corresponding to the feature points of the eyes in the light array102. In this case, the AR device 1000 may calculate the coordinate valueof the pupil feature point and the coordinate value of the glint featurepoint of the eyes calibrated by reflecting the degree of bias of thetemple 191 of the AR device 1000 and/or the degree of bias of the nosesupport 192. The calibrated coordinate values may be input to a mappingfunction.

In operation S1225, the AR device 1000 may identify the pupil positionof the user's eyes based on the IR light reflected from the lightreflector 1400. For example, when the light receiver 1520 is an IRcamera, the AR device 1000 may identify the pupil position of the user'seyes within an image captured by the IR camera from the image.Alternatively, for example, when the light receiver 1520 is an IRdetector, the AR device 1000 may analyze the IR light sequentiallyobtained by the IR detector, thereby calculating the pupil position ofthe user's eyes. The AR device 1000 may identify the pupil central pointof the user's eyes, thereby identifying the pupil position of the user'seyes.

In operation S1230, the AR device 1000 may obtain a gaze direction ofthe user. The AR device 1000 may calculate a position of the center ofthe user's eyes. The center of the user's eyes may be the center ofuser's eyeballs. The AR device 1000 may calculate the position of thecenter of the user's eyes, based on the pupil position of the user'seyes and the degree of bias of the support 190. For example, theprocessor 1800 may calculate the position of the center of the user'seyes so that a value calculated based on a matrix for calibrating thedegree of bias of the support 190, a value indicating the position ofthe center of the user's eyes and a bias of axis of the image obtainedby capturing the user's eyes can be a value of the pupil position of theuser's eyes obtained by the pupil position detection code 1760. Forexample, the center of the eye may be the center of the eyeball, and theposition of the center of the user's eyes may have a 3D coordinate valuein a coordinate system of a real space.

The AR device 1000 may calculate a position of the gaze point of theuser. In order to calculate the position of the gaze point of the user,the AR device 1000 may previously generate a mapping function forcalculating the position of the gaze point from features of the user'seyes. The mapping function is a function for calculating the position ofthe gaze point of the user in consideration of features of the user'seyes and bias information of the support 190, and may be generatedduring a calibration process of the calibration code 1780. For example,the position of the gaze point may have a 3D coordinate value in thecoordinate system in the real space, but the disclosure is not limitedthereto. For example, the position of the gaze point may have acoordinate value in the coordinate system of the waveguide 170, but isnot limited thereto.

The AR device 1000 may calibrate the features related to the user's gazebased on the degree of bias obtained from the bias determination code1750. Also, the AR device 1000 may apply the features related to theuser's gaze calibrated based on the degree of bias to the mappingfunction, thereby calculating the position of the gaze point of theuser. Also, a gaze direction of the user may be determined based on theposition of the central point of the user's eyes and the gaze point ofthe user calculated by the gaze determination code 1770.

Meanwhile, the AR device 1000 may calibrate the mapping function basedon the bias angle of the support 190. The AR device 1000 may calibratethe mapping function for obtaining the gaze point of the user based on adefault bias angle and features of eyes.

For example, when the light receiver 1520 is an IR camera, the AR device1000 may display a target point for calibration through thewaveguide170, and capture the user's eyes looking at the target point byusing the IR camera. In addition, the processor 1800 may identify andanalyze a pattern within the image obtained by capturing the user'seyes, thereby obtaining a bias angle of the support 190. In addition,the AR device 1000 may detect positions of the feature points related tothe user's eyes from the image obtained by capturing the user's eyes,and input the positions of the feature points of the user's eyes and thebias angle of the support 190 into the mapping function. The AR device1000 may calibrate the mapping function so that a position value of thetarget point may be output from the mapping function to which thepositions of the feature points of the user's eyes and the bias angle ofthe support 190 are input.

For example, when the light receiver 1520 is an IR detector, the ARdevice 1000 may display the target point for calibration on thewaveguide170 and control the IR scanner to emit IR light for scanningthe user's eyes looking at the target point. In addition, the AR device1000 may receive and analyze IR light reflected from the user's eyesthrough the IR detector, identify a pattern of the light reflector 1400,and estimate the bias angle of the temple 191. The AR device 1000analyze the IR light based on the bias angle of the temple 191, therebyidentifying positions of the calibrated feature points of eyes. Forexample, when the AR device 1000 estimates the positions of the featurepoints of eyes by using the IR scanner and the IR detector, becauseresults of estimating the positions of the feature points of eyes areaffected by an operating angle of the IR scanner, the positions of thefeature points of eyes may be calibrated based on the bias of thesupport 190 by calculating a value obtained by subtracting the biasangle of the support 190 from the operating angle of the IR scanner.

In addition, the AR device 1000 may input the positions of thecalibrated feature points of the eyes into the mapping function, andcalibrate the mapping function so that the position value of the targetpoint may be output from the mapping function in which the calibratedpositions of the feature points of the eyes are input.

An example embodiment of the disclosure may be implemented as arecording medium including computer-readable instructions such as acomputer-executable program module. According to an example embodiment,the computer-readable instructions may be computer codes, which areimplemented as one or more computer-executable program modules. Acomputer-readable medium may be any available medium which is accessibleby a computer, and may include a volatile or non-volatile medium and aremovable or non-removable medium. Also, the computer-readable mediummay include computer storage medium and communication medium. Thecomputer storage media include both volatile and non-volatile, removableand non-removable media implemented in any method or technique forstoring information such as computer readable instructions, datastructures, program codes, program modules or other data. Thecommunication medium may typically include computer-readableinstructions, data structures, or other data of a modulated data signalsuch as program modules.

A computer-readable storage medium may be provided in a form of anon-transitory storage medium. Here, the term ‘non-transitory storagemedium’ refers to a tangible device and does not include a signal (e.g.,an electromagnetic wave), and does not distinguish between a case wheredata is stored in a storage medium semi-permanently and a case wheredata is stored temporarily. For example, the non-transitory storagemedium may include a buffer in which data is temporarily stored.

According to an example embodiment of the disclosure, the methodaccording to various embodiments of the disclosure disclosed herein maybe included in a computer program product and provided. The computerprogram product may be traded between a seller and a purchaser as acommodity. The computer program product may be distributed in a form ofa machine-readable storage medium (e.g., compact disk read only memory(CD-ROM)), or may be distributed online (e.g., downloaded or uploaded)through an application store (e.g., PlayStore™) or directly between twouser devices (e.g., smart phones). In the case of online distribution,at least a portion of the computer program product (e.g., a downloadableapp) may be temporarily stored in a machine-readable storage medium suchas a manufacturer's server, an application store's server, or a memoryof a relay server.

In addition, in the specification, the term “unit” may be a hardwarecomponent such as a processor or a circuit, and/or a software componentexecuted by a hardware component such as a processor.

Also, in the specification, the expression “include at least one of a, bor c” means “include only a”, “include only b”, “include only c”,“include a and b”, “include b and c”, “include a and c”, or “include a,b, and c”.

The above-described description of the disclosure is provided only forillustrative purposes, and those of skill in the art will understandthat the disclosure may be easily modified into other detailedconfigurations without modifying technical aspects and essentialfeatures of the disclosure. Therefore, it should be understood that theabove-described embodiments are exemplary in all respects and are notlimited. For example, the elements described as single entities may bedistributed in implementation, and similarly, the elements described asdistributed may be combined in implementation.

The scope of the disclosure is not defined by the detailed descriptionof the disclosure but by the following claims, and all modifications oralternatives derived from the scope and spirit of the claims andequivalents thereof fall within the scope of the disclosure.

1. An augmented reality (AR) device comprising: a waveguide; a lightreflector comprising a pattern; a support configured to fix the ARdevice to a user's face of the AR device; a light emitter and a lightreceiver installed on the support; and at least one processor configuredto: control the light emitter to emit light toward the light reflector,identify the pattern based on the light received by the light receiver,and obtain gaze information of a user of the AR device based on theidentified pattern, wherein the light emitted toward the light reflectoris reflected by the light reflector and directed toward an eye of theuser, and wherein the light received by the light receiver compriseslight from the light directed toward the eye of the user being reflectedby the eye of the user.
 2. The AR device of claim 1, wherein the supportcomprises: a temple extending from a frame around the waveguide to bepositioned on an ear of the user; and a nose support extending from theframe and positioned on a nose of the user.
 3. The AR device of claim 2,wherein the light reflector is formed on the waveguide.
 4. The AR deviceof claim 3, wherein the at least one processor is further configured toanalyze the identified pattern and identify a degree of bias of thesupport with respect to the frame, the support extending from the frame.5. The AR device of claim 4, wherein the at least one processor isfurther configured to: generate a mapping function for calculating aposition of a gaze point of the user based on the degree of bias of thesupport with respect to the frame, and based on the mapping function andthe degree of bias of the support with respect to the frame, obtain thegaze information of the user.
 6. The AR device of claim 5, wherein theat least one processor is further configured to, based on the lightreceived by the light receiver, obtain a position of one or more featurepoints corresponding to the eye of the user.
 7. The AR device of claim6, wherein the at least one processor is configured to input theposition of the one or more feature points corresponding to the eye ofthe user and the degree of bias of the support with respect to the frameinto the mapping function and calculate the position of the gaze pointof the user.
 8. The AR device of claim 5, wherein the at least oneprocessor is further configured to: display a target point at a specificposition on the waveguide in order to calibrate the mapping function,receive light reflected by the eye of the user looking at the displayedtarget point through the light receiver, and calibrate the mappingfunction based on the light reflected by the eye of the user looking atthe displayed target point.
 9. The AR device of claim 8, wherein, forcalibration of the mapping function, the at least one processor isfurther configured to: based on the light reflected by the eye of theuser looking at the displayed target point, identify the pattern of thelight reflector, based on the identified pattern, identify the degree ofbias of the support, and based on the light reflected by the eye of theuser looking at the displayed target point, obtain a position of one ormore feature points corresponding to the eye of the user looking at thedisplayed target point.
 10. The AR device of claim 9, wherein the atleast one processor is further configured to: input a degree of bias ofthe temple and the position of the one or more feature points into themapping function, and calibrate the mapping function so that a positionvalue of the target point is output from the mapping function.
 11. Amethod, performed by an augmented reality (AR) device, of detecting auser's gaze, the method comprising: emitting, by a light emitterinstalled in a support of the AR device, light toward a light reflectorcomprising a pattern, the light emitted by the light emitter beingdirected toward an eye of a user wearing the AR device; receiving, by alight receiver installed on the support, the light reflected by the eyeof the user; identifying the pattern based on the light received throughthe light receiver; and obtaining gaze information of the user based onthe identified pattern.
 12. The method of claim 11, further comprisinganalyzing the identified pattern and identifying a degree of bias of thesupport with respect to the frame based on the identified pattern,wherein the support extends from the frame, wherein the obtaining of thegaze information comprises determining a gaze direction of the userbased on the degree of bias of the support with respect to the frame.13. The method of claim 12, further comprising generating a mappingfunction for calculating a position of a gaze point of the user based onthe degree of bias of the support with respect to the frame, wherein theobtaining of the gaze information comprises calculating the position ofthe gaze point of the user based on the mapping function and the degreeof bias of the support with respect to the frame.
 14. The method ofclaim 13, further comprising, based on the light received by the lightreceiver, obtaining a position of one or more feature pointscorresponding to the eye of the user, wherein the obtaining of the gazeinformation comprises inputting the position of the one or more featurepoints corresponding to the eye of the user and the degree of bias ofthe support with respect to the frame into the mapping function andcalculating the position of the gaze point of the user.
 15. Acomputer-readable recording medium having recorded thereon a program forexecuting the method of claim 11 on a computer.